A hybrid and cross-protocol architecture with semantics and syntax awareness to improve intrusion detection efficiency in Voice over IP environments

Includes abstract.Includes bibliographical references (leaves 134-140).Voice and data have been traditionally carried on different types of networks based on different technologies, namely, circuit switching and packet switching respectively. Convergence in networks enables carrying voice, video, and other data on the same packet-switched infrastructure, and provides various services related to these kinds of data in a unified way. Voice over Internet Protocol (VoIP) stands out as the standard that benefits from convergence by carrying voice calls over the packet-switched infrastructure of the Internet. Although sharing the same physical infrastructure with data networks makes convergence attractive in terms of cost and management, it also makes VoIP environments inherit all the security weaknesses of Internet Protocol (IP). In addition, VoIP networks come with their own set of security concerns. Voice traffic on converged networks is packet-switched and vulnerable to interception with the same techniques used to sniff other traffic on a Local Area Network (LAN) or Wide Area Network (WAN). Denial of Service attacks (DoS) are among the most critical threats to VoIP due to the disruption of service and loss of revenue they cause. VoIP systems are supposed to provide the same level of security provided by traditional Public Switched Telephone Networks (PSTNs), although more functionality and intelligence are distributed to the endpoints, and more protocols are involved to provide better service. A new design taking into consideration all the above factors with better techniques in Intrusion Detection are therefore needed. This thesis describes the design and implementation of a host-based Intrusion Detection System (IDS) that targets VoIP environments. Our intrusion detection system combines two types of modules for better detection capabilities, namely, a specification-based and a signaturebased module. Our specification-based module takes the specifications of VoIP applications and protocols as the detection baseline. Any deviation from the protocol’s proper behavior described by its specifications is considered anomaly. The Communicating Extended Finite State Machines model (CEFSMs) is used to trace the behavior of the protocols involved in VoIP, and to help exchange detection results among protocols in a stateful and cross-protocol manner. The signature-based module is built in part upon State Transition Analysis Techniques which are used to model and detect computer penetrations. Both detection modules allow for protocol-syntax and protocol-semantics awareness. Our intrusion detection uses the aforementioned techniques to cover the threats propagated via low-level protocols such as IP, ICMP, UDP, and TCP

Removing Ambiguities of IP Telephony Traffic Using Protocol Scrubbers

Network intrusion detection systems (NIDSs) face the serious challenge of attacks such as insertion and evasion attacks that are caused by ambiguous network traffic. Such ambiguity comes as a result of the nature of network traffic which includes protocol implementation variations and errors alongside legitimate network traffic. Moreover, attackers can intentionally introduce further ambiguities in the traffic. Consequently, NIDSs need to be aware of these ambiguities when detection is performed and make sure to differentiate between true attacks and protocol implementation variations or errors; otherwise, detection accuracy can be affected negatively. In this paper we present the design and implementation of tools that are called protocol scrubbers whose main functionality is to remove ambiguities from network traffic before it is presented to the NIDS. The proposed protocol scrubbers are designed for session initiation and data transfer protocols in IP telephony systems. They guarantee that the traffic presented to NIDSs is unambiguous by eliminating ambiguous behaviors of protocols using well-designed protocol state machines, and walking through packet headers of protocols to make sure packets will be interpreted in the desired way by the NIDS. The experimental results shown in this paper demonstrate the good quality and applicability of the introduced scrubbers

Enhancing the detection of metamorphic malware using call graphs

Malware stands for malicious software. It is software that is designed with a harmful intent. A malware detector is a system that attempts to identify malware using Application Programming Interface (API) call graph technique and/or other techniques. API call graph techniques follow two main steps, namely, transformation of malware samples into an API call graph using API call graph construction algorithm, and matching the constructed graph against existing malware call graph samples using graph matching algorithm. A major issue facing malware API call graph construction algorithms is building a precise call graph from information collected about malware samples. On the other hand call graph matching is an NP-complete problem and is slow because of computational complexity. In this study, a malware detection system based on API call graph is proposed. In the proposed system, each malware sample is represented as an API call graph. API call graph construction algorithm is used to transform input malware samples into API call graph by integrating API calls and operating system resource to represent graph nodes. Moreover, the dependence between different types of nodes is identified and represented using graph edges. After that, graph matching algorithm is used to calculate similarity between the input sample and malware API call graph samples that are stored in a database. The graph matching algorithm is based on an enhanced graph edit distance algorithm that simplifies the computational complexity using a greedy approach to select best common subgraphs from the integrating API call graph with high similarity, which helps in terms of detecting metamorphic malware. Experimental results on 514 malware samples demonstrate that the proposed system has 98% accuracy and 0 false positive rates. Detailed comparisons against other detection methods have been carried out and significant improvement over them is shown